# Efficient pre-training

An 11-billion parameter large language model trained with the innovative SuperBPE tokenizer, supporting superword unit recognition and subword tokenization capabilities.

mGTE series multilingual text encoder, supporting 75 languages, with a maximum context length of 8192, based on BERT+RoPE+GLU architecture, excelling in GLUE and XTREME-R benchmarks

A BERT variant trained on the 100MW BabyLM Challenge dataset, optimized for performance on medium-scale corpora

This is an 80M parameter M2-BERT pre-trained checkpoint with a sequence length of 2048, fine-tuned for long-context retrieval tasks.

A BERT-small model pre-trained on Czech web corpora using the RetroMAE objective, developed by Seznam.cz, suitable for various natural language processing tasks.

Sheared-LLaMA-1.3B is an efficient language model obtained through structured pruning and continual pre-training based on LLaMA-2-7B

This model investigates the effectiveness of masking 15% of content in masked language modeling, employing a pre-layer normalization approach.

DistilBERT is a lightweight distilled version of BERT, retaining most of the performance but with fewer parameters.

Russian pre-trained Transformer language model developed by SberDevices team, based on GPT2 architecture, supports 1024 sequence length, trained on 80 billion tokens.

A RoBERTa base model trained using the WECHSEL method, specifically optimized for Swahili to achieve efficient cross-lingual transfer.